Designing and Optimizing Storage in an Exchange Server 2007 Environment (part 2) - Designing the Right Data Storage Structure for Exchange Server 2007

Choosing the Right Connectivity for NAS

All the high-speed disks in the world won’t amount to much if you can’t get the data to and from the Exchange servers quickly. In a NAS environment, the network itself is the biggest concern for performance. Most NAS devices on the market use very fast heads that are literally dedicated computers with high-performance processors and loads of memory. With SCSI RAID controllers on board, they can easily saturate multiple 100-Mb Ethernet connections. Attaching such a device to a low-end switch would result in the NAS running in an extremely restricted manner. Strongly consider using a switch that will enable you to use a gigabit connection.

Consider creating a separate network for the NAS environment. Suppose, for example, that the NAS is going to support a number of Exchange servers. By multihoming the Exchange servers, one Ethernet connection can face the users and provide connectivity to the mail clients, whereas the other interface can be dedicated to NAS traffic. This allows each interface to run unfettered by the traffic associated with the other network. This also enables you to upgrade only a subset of the network to improve performance and save money. The traffic of the database transaction back to the NAS device by Exchange would be much greater than the traffic associated with users viewing their mail because the traffic that would normally go to the local disk would now be traveling across the Ethernet via the virtual disk driver that connects the NAS to the Exchange server.

When selecting network gear for a NAS out-of-band network, focus on packets per second. Whenever possible, build this NAS network with multiple switches that are cross-connected. Connect each server to both switches with the NICs in a Teamed mode. This not only adds bandwidth, but also creates redundancy for the Network layer. Odds are if the application warranted the use of a NAS device, it deserves redundancy at the network level as well.

When selecting NICs for the servers, strongly consider the use of NICs that support Transmission Control Protocol (TCP) offload processing. This means that the work involved with network transfers is performed by the NIC itself rather then increasing the load on the server’s CPUs. Because the NIC is designed with data transfer in mind, the result is the ability to move huge amounts of data without impacting the overall performance of the Exchange server. Because network overhead is associated with mounting NAS disks, this type of configuration can be very helpful for the Exchange server.

Choosing the Right Connectivity for SANs

When attaching. to a SAN, you will be using HBAs via Fibre Channel rather than NICs via Ethernet. HBAs can be relatively expensive, but they offer much greater throughput than NICs and NAS would offer. Between the higher speeds (4Gb for Fibre Channel versus 1Gb for Ethernet) and the lower overhead involved in the protocol, a HBA-attached SAN can move significantly more data in the same period of time. This can be especially useful in situations where a large number of disks are being accessed.

SANs are generally attached to the HBAs via a Fibre Channel fabric. A Fibre Channel fabric is created by a set of interconnected HBAs, bridges, storage devices, and switches. Strongly consider implementing multiple fabrics for redundancy. Generally, a fabric can be thought of as a set of switches sharing interswitch links along with the devices to which they are connected. A SAN with multiple switches not connected by interswitch links provides multiple fabrics.

The SAN connects to the switch fabric through controllers. These controllers are what combine the disks together into larger aggregates and servers as the entry and exit point for data. SAN controllers generally contain very large caches of memory (typically 2–4GB) to improve performance. Multiple controllers are always recommended for redundancy and performance.

When thinking about the connectivity between the Exchange servers and the SAN, always try to use multiple LUNs and connect them such that half the LUNs prefer Controller A and half prefer Controller B. This helps even out the load across the controllers and increases overall throughput of the SAN. In the event of controller failure or controller maintenance, the connectivity is picked up by the remaining controller.

When planning your SAN storage, be very aware of how your particular SAN and switch fabric deal with zoning. The concept of zoning is similar to the concept of virtual LANs (VLANs) in networking. The objective is to ensure that only the necessary servers can see the disks that will be provisioned to them. Depending on your particular solution, this is performed via LUN masking, hard/soft zoning, port zoning, or through the use of worldwide names. These concepts work as follows:

• LUN masking— LUN masking is a process that makes particular LUNs available to some hosts but not to others. This process is akin to setting permissions on a resource to determine which hosts are allowed to access them. This is particularly important in Windows environments where a server will attempt to write a signature to a newly discovered disk. This can render an existing LUN unavailable to its originally intended host.

• Hard/soft zoning— In this context, hard and soft refer to the location of the implementation of this type of zoning. Hard zoning is done at a hardware level and soft zoning is done in software. Hard zoning physically blocks access to a zone from any device outside of the zone. Soft zoning uses filters in the switch fabric that prevent ports from being seen from outside of their assigned zones.

• Port zoning— Port zoning uses physical ports to define security zones. A user’s access to data is determined by what physical port he is connected to. The drawback with port zoning is that zone information must be updated every time a user changes switch ports. In addition, port zoning does not allow zones to overlap. Port zoning is normally implemented using hard zoning, but can also be implemented using soft zoning.

• World Wide Name (WWN) zoning— WWN zoning uses name servers in the switches to either allow or disallow access to particular WWNs in the fabric. A major advantage of WWN zoning is the ability to modify the fabric without having to redo the zone information. SAN-related devices like HBAs are built with unique WWNs installed into them not unlike Media Access Control (MAC) addresses in network interfaces.

Choosing the Right Type of Disks

When researching SAN and NAS devices, you will discover that you have several types of disks available to you. These disks will vary by architecture (SCSI versus ATA versus Fibre Channel) as well as by size. Current disks are available in sizes ranging from 72GB to 250GB each.

In terms of size, your decisions will be based on three factors:

• Price

• Capacity

• Performance

Generally speaking, the larger the disk, the more you pay for it. Capacity refers to the total amount of space you plan to deploy. If, for example, you needed to deploy 2TB of space, you could use eight 250-GB disks or thirty-two 72-GB disks. Why would you pick one configuration over the other?

If you opted to use eight 250-GB disks, you’d be using less capacity on your SAN or NAS device. If you expected to expand capacity in the future, you’d be able to expand further before needing to purchase additional disk shelves or chassis. The potential downside to this approach is that eight 250-GB disks might be more expensive than thirty-two 72-GB disks. The other more noticeable impact is in the area of I/O performance. Assuming the spindle speeds were the same for both disks, you would get four times more I/O out of the thirty-two 72-GB disks than you would from the eight 250-GB disks. Depending on whether your application needed the additional I/O, this might be a deciding factor.

Useful to note is that with sufficient memory in an Exchange 2007 server, disk I/O requirements are roughly one fourth what they were in an Exchange 2003 server with the same number of users. This behavior was specifically engineered into Exchange 2007 to take advantage of the ever-increasing capacity of hard disks. Hard disk capacity is increasing drastically every year with nearly no improvements in I/O performance. According to Seagate, although disk capacity increased 15,000 times from 1987 to 2004, the random I/O performance increased only 11 times during the same period.

In addition to choosing the size of the disks you deploy, you also have a choice in terms of the disk architecture. Your most common choices are as follows:

• Serial ATA (SATA)

• SCSI

• Fibre Channel

SATA is generally the least-expensive option. SATA disks provide excellent throughput, nearly equal to SCSI, at a much better price. High-capacity disks are usually available as SATA first because it is a more common market for disks. Newer implementations of SATA include high-performance functions such as command queuing, which give them performance that approaches that of SCSI.

SCSI disks have been around for decades. It’s a very well-proven technology and is known for having very high performance as well as very high reliability. SCSI disks are less expensive than Fibre Channel disks but offer lower throughput through the bus. This results in needing more controllers to manage the disks themselves and lower performance than Fibre Channel disks.

Fibre Channel disks are the highest-performance drives available today. They are also the most expensive and generally trail a full generation behind other formats in terms of capacity. If performance is your number-one concern, the Fibre Channel disk can’t be beat.

Slicing and Dicing the Available Disk

Simple physics tells you that you’ll get improvements in performance as you add more disks to an array. Because each drive’s read/write head can operate simultaneously, you get a fairly linear improvement as drives are added. NAS and SAN offer the advantage of dynamically increasing the size of a volume without taking the volume offline. This allows for the addition of even more spindles.

Although it’s possible to later resize a volume from a NAS or SAN, you must be careful not to oversubscribe the device. Devices that support snapshots of the data reserve twice the volume size that they claim for capacity. So, to make 100GB available to a server, the NAS reserves 200GB on itself. This ensures that it is able to complete all transactions. This function can be disabled on most devices, but it is not recommended. This removes the protection from oversubscription of the disks.

When provisioning disk space for an Exchange server, you should consider a few rules of thumb when optimizing performance.

In a perfect world, an entire SAN or NAS would be dedicated to just the Exchange 2007 environment. This would reduce the possibility of contention with other applications. If your budget doesn’t allow for this, be very aware of what applications are being shared with your SAN or NAS.

If you can’t dedicate a SAN or NAS to your Exchange environment, build your aggregate from disks that are spread out across multiple shelves. This helps distribute the load across multiple backplanes and results in fewer spikes in performances.

Try not to make LUNs larger than they need to be. For example, if you plan to have four storage groups with 50GB of mail each, create four LUNs of 50GB each rather than a single LUN of 200GB. This allows you to separate the LUNs across both controllers and improves the performance of the system. The potential pitfall here is that you could run out of drive letters because Exchange 2007 allows for up to 50 databases in up to 50 storage groups in the Enterprise Edition. To work around this, mount the LUNs as mount points instead of drive letters. This can greatly simplify expansions of Exchange 2007 servers as you can place a storage group on a drive letter and then mount new LUNs as mount points for each new database that you need to bring online. This is exceptionally useful when using snapshot functions in NAS or SAN where the database has to be dismounted for an integrity check because this typically occurs at the LUN level.

To mount a LUN as a mount point rather than a drive letter, perform the following steps:

Predicting Disk Performance with Exchange Server 2007

When planning the number of disks to use for LUNs for various functions in Exchange 2007, the question that invariably comes up is “How many spindles do I need for good performance?” Although it is fairly straightforward to determine the I/O needs for various functions in Exchange 2007, it can be trickier to predict the effect that the disk configuration will have on the system. One of the most common configurations is to utilize RAID 5 to provide redundancy at the disk level. To understand the impact of RAID 5, consider the following:

RAID-5 performance can be approximated as %Reads*IOPS per disk*(disks-1))+(%Writes*IOPS per disk*((disks-1)/4))= Total IOPS

Or, for the more mathematically oriented:

Total IOPS = (R*I(d-1))+(W*I((d-1)/4))

where:

• R = % Reads

• W = % Writes

• I = Input / Output operations per second (IOPS) per disk

• d = number of disks in RAID5

• T = Total IOPS

With typical IOPS performance per disk being:

• 140-150 Random IOPS from 15,000-RPM disks

• 100-120 Random IOPS from 10,000-RPM disks

• 75-100 Random IOPS from 7,200-RPM disks